In [1]:
import open3d as o3d
import numpy as np
import matplotlib.pyplot as plt
import os
import sys

# only needed for tutorial, monkey patches visualization
sys.path.append('..')
import open3d_tutorial as o3dtut
# change to True if you want to interact with the visualization windows
o3dtut.interactive = not "CI" in os.environ

Surface Reconstruction

In many scenarios we want to generate a dense 3D geometry, i.e., a triangle mesh. However, from a multi-view stereo method, or a depth sensor we only obtain an unstructured point cloud. To get a triangle mesh from this unstructured input we need to perform surface reconstruction. In the literature there exists a couple of methods and Open3D currently implements the following:

Alpha shapes

The alpha shape [Edelsbrunner1983] is a generalization of a convex hull. As described here one can intuitively think of an alpha shape as the following: Imagine a huge mass of ice cream containing the points $S$ as hard chocolate pieces. Using one of these sphere-formed ice cream spoons we carve out all parts of the ice cream block we can reach without bumping into chocolate pieces, thereby even carving out holes in the inside (e.g., parts not reachable by simply moving the spoon from the outside). We will eventually end up with a (not necessarily convex) object bounded by caps, arcs and points. If we now straighten all round faces to triangles and line segments, we have an intuitive description of what is called the alpha shape of $S$.

Open3D implements the method create_from_point_cloud_alpha_shape that involves the tradeoff parameter alpha.

In [2]:
mesh = o3dtut.get_bunny_mesh()
pcd = mesh.sample_points_poisson_disk(750)
o3d.visualization.draw_geometries([pcd])
alpha = 0.03
print(f"alpha={alpha:.3f}")
mesh = o3d.geometry.TriangleMesh.create_from_point_cloud_alpha_shape(pcd, alpha)
mesh.compute_vertex_normals()
o3d.visualization.draw_geometries([mesh], mesh_show_back_face=True)

alpha=0.030

The implementation is based on the convex hull of the point cloud. If we want to compute multiple alpha shapes from a given point cloud, then we can save some computation by only computing the convex hull once and pass it to create_from_point_cloud_alpha_shape.

In [3]:
tetra_mesh, pt_map = o3d.geometry.TetraMesh.create_from_point_cloud(pcd)
for alpha in np.logspace(np.log10(0.5), np.log10(0.01), num=4):
    print(f"alpha={alpha:.3f}")
    mesh = o3d.geometry.TriangleMesh.create_from_point_cloud_alpha_shape(
        pcd, alpha, tetra_mesh, pt_map)
    mesh.compute_vertex_normals()
    o3d.visualization.draw_geometries([mesh], mesh_show_back_face=True)

alpha=0.500

alpha=0.136

alpha=0.037

alpha=0.010
In [5]:
mesh = o3d.io.read_triangle_mesh("../../test_data/my test data/model_mesh.ply")
pcd = o3d.io.read_point_cloud("../../test_data/my test data/050.ply")
o3d.visualization.draw_geometries([pcd])
alpha = 0.07
print(f"alpha={alpha:.3f}")
mesh = o3d.geometry.TriangleMesh.create_from_point_cloud_alpha_shape(pcd, alpha)
mesh.compute_vertex_normals()
o3d.visualization.draw_geometries([mesh], mesh_show_back_face=True)

alpha=0.090
In [6]:
tetra_mesh, pt_map = o3d.geometry.TetraMesh.create_from_point_cloud(pcd)
for alpha in np.logspace(np.log10(0.5), np.log10(0.01), num=4):
    print(f"alpha={alpha:.3f}")
    mesh = o3d.geometry.TriangleMesh.create_from_point_cloud_alpha_shape(
        pcd, alpha, tetra_mesh, pt_map)
    mesh.compute_vertex_normals()
    o3d.visualization.draw_geometries([mesh], mesh_show_back_face=True)

alpha=0.500

alpha=0.136

alpha=0.037

alpha=0.010

Ball pivoting

The ball pivoting algorithm (BPA) [Bernardini1999] is a surface reconstruction method which is related to alpha shapes. Intuitively, think of a 3D ball with a given radius that we drop on the point cloud. If it hits any 3 points (and it does not fall through those 3 points) it creates a triangles. Then, the algorithm starts pivoting from the edges of the existing triangles and every time it hits 3 points where the ball does not fall through we create another triangle.

Open3D implements this method in create_from_point_cloud_ball_pivoting. The method accepts a list of radii as parameter that corresponds to the radii of the individual balls that are pivoted on the point cloud.

**Note:** This algorithm assumes that the `PointCloud` has `normals`.
In [7]:
gt_mesh = o3dtut.get_bunny_mesh()
gt_mesh.compute_vertex_normals()
pcd = gt_mesh.sample_points_poisson_disk(3000)
o3d.visualization.draw_geometries([pcd])
In [8]:
radii = [0.005, 0.01, 0.02, 0.04]
rec_mesh = o3d.geometry.TriangleMesh.create_from_point_cloud_ball_pivoting(
    pcd, o3d.utility.DoubleVector(radii))
o3d.visualization.draw_geometries([pcd, rec_mesh])
In [9]:
pcd = o3d.io.read_point_cloud("../../test_data/my test data/050.ply")
o3d.visualization.draw_geometries([pcd])
In [10]:
radii = [0.005, 0.01, 0.02, 0.04]
rec_mesh = o3d.geometry.TriangleMesh.create_from_point_cloud_ball_pivoting(
    pcd, o3d.utility.DoubleVector(radii))
o3d.visualization.draw_geometries([pcd, rec_mesh])

Poisson surface reconstruction

The Poisson surface reconstruction method [Kazhdan2006] solves a regularized optimization problem to obtain a smooth surface. For this reason, Poisson surface reconstruction can be preferable to the methods mentioned above, as they produce non-smooth results since the points of the PointCloud are also the vertices of the resulting triangle mesh without any modifications.

Open3D implements the method create_from_point_cloud_poisson which is basically a wrapper of the code of Kazhdan. An important parameter of the function is depth that defines the depth of the octree used for the surface reconstruction and hence implies the resolution of the resulting triangle mesh. A higher depth value means a mesh with more details.

**Note:** This algorithm assumes that the `PointCloud` has `normals`.
In [11]:
pcd = o3dtut.get_eagle_pcd()
print(pcd)
o3d.visualization.draw_geometries([pcd],
                                  zoom=0.664,
                                  front=[-0.4761, -0.4698, -0.7434],
                                  lookat=[1.8900, 3.2596, 0.9284],
                                  up=[0.2304, -0.8825, 0.4101])

geometry::PointCloud with 796825 points.
In [12]:
print('run Poisson surface reconstruction')
with o3d.utility.VerbosityContextManager(
        o3d.utility.VerbosityLevel.Debug) as cm:
    mesh, densities = o3d.geometry.TriangleMesh.create_from_point_cloud_poisson(
        pcd, depth=9)
print(mesh)
o3d.visualization.draw_geometries([mesh],
                                  zoom=0.664,
                                  front=[-0.4761, -0.4698, -0.7434],
                                  lookat=[1.8900, 3.2596, 0.9284],
                                  up=[0.2304, -0.8825, 0.4101])

run Poisson surface reconstruction
[Open3D DEBUG] Input Points / Samples: 796825 / 368254
[Open3D DEBUG] #   Got kernel density: 0.102381 (s), 250.52 (MB) / 250.52 (MB) / 263 (MB)
[Open3D DEBUG] #     Got normal field: 0.45385 (s), 353.773 (MB) / 353.773 (MB) / 353 (MB)
[Open3D DEBUG] Point weight / Estimated Area: 2.623550e-06 / 2.090510e+00
[Open3D DEBUG] #       Finalized tree: 0.566193 (s), 484.484 (MB) / 484.484 (MB) / 484 (MB)
[Open3D DEBUG] #  Set FEM constraints: 1.04521 (s), 444.812 (MB) / 484.484 (MB) / 484 (MB)
[Open3D DEBUG] #Set point constraints: 0.203878 (s), 444.812 (MB) / 484.484 (MB) / 484 (MB)
[Open3D DEBUG] Leaf Nodes / Active Nodes / Ghost Nodes: 2945433 / 3365000 / 1209
[Open3D DEBUG] Memory Usage: 444.812 MB
[Open3D DEBUG] # Linear system solved: 1.4974 (s), 487.391 (MB) / 487.391 (MB) / 487 (MB)
[Open3D DEBUG] Got average: 0.051687 (s), 447.977 (MB) / 487.391 (MB) / 487 (MB)
[Open3D DEBUG] Iso-Value: 5.028479e-01 = 4.006817e+05 / 7.968250e+05
[Open3D DEBUG] #          Total Solve:       9.9 (s),     633.1 (MB)
geometry::TriangleMesh with 563112 points and 1126072 triangles.

Poisson surface reconstruction will also create triangles in areas of low point density, and even extrapolates into some areas (see bottom of the eagle output above). The create_from_point_cloud_poisson function has a second densities return value that indicates for each vertex the density. A low density value means that the vertex is only supported by a low number of points from the input point cloud.

In the code below we visualize the density in 3D using pseudo color. Violet indicates low density and yellow indicates a high density.

In [13]:
print('visualize densities')
densities = np.asarray(densities)
density_colors = plt.get_cmap('plasma')(
    (densities - densities.min()) / (densities.max() - densities.min()))
density_colors = density_colors[:, :3]
density_mesh = o3d.geometry.TriangleMesh()
density_mesh.vertices = mesh.vertices
density_mesh.triangles = mesh.triangles
density_mesh.triangle_normals = mesh.triangle_normals
density_mesh.vertex_colors = o3d.utility.Vector3dVector(density_colors)
o3d.visualization.draw_geometries([density_mesh],
                                  zoom=0.664,
                                  front=[-0.4761, -0.4698, -0.7434],
                                  lookat=[1.8900, 3.2596, 0.9284],
                                  up=[0.2304, -0.8825, 0.4101])

visualize densities

We can further use the density values to remove vertices and triangles that have a low support. In the code below we remove all vertices (and connected triangles) that have a lower density value than the $0.01$ quantile of all density values.

In [14]:
print('remove low density vertices')
vertices_to_remove = densities < np.quantile(densities, 0.01)
mesh.remove_vertices_by_mask(vertices_to_remove)
print(mesh)
o3d.visualization.draw_geometries([mesh],
                                  zoom=0.664,
                                  front=[-0.4761, -0.4698, -0.7434],
                                  lookat=[1.8900, 3.2596, 0.9284],
                                  up=[0.2304, -0.8825, 0.4101])

remove low density vertices
geometry::TriangleMesh with 557480 points and 1113210 triangles.
In [ ]:
In [26]:
pcd = o3d.io.read_point_cloud("../../test_data/my test data/050.ply")
print(pcd)
o3d.visualization.draw_geometries([pcd],
                                  zoom=0.664,
                                  front=[-0.4761, -0.4698, -0.7434],
                                  lookat=[1.8900, 3.2596, 0.9284],
                                  up=[0.2304, -0.8825, 0.4101])

geometry::PointCloud with 26405 points.
In [27]:
print('run Poisson surface reconstruction')
with o3d.utility.VerbosityContextManager(
        o3d.utility.VerbosityLevel.Debug) as cm:
    mesh, densities = o3d.geometry.TriangleMesh.create_from_point_cloud_poisson(
        pcd, depth=9)
print(mesh)
o3d.visualization.draw_geometries([mesh],
                                  zoom=0.664,
                                  front=[-0.4761, -0.4698, -0.7434],
                                  lookat=[1.8900, 3.2596, 0.9284],
                                  up=[0.2304, -0.8825, 0.4101])

run Poisson surface reconstruction
[Open3D DEBUG] Input Points / Samples: 26405 / 24755
[Open3D DEBUG] #   Got kernel density: 0.0293889 (s), 474.375 (MB) / 633.051 (MB) / 695 (MB)
[Open3D DEBUG] #     Got normal field: 0.0529389 (s), 474.375 (MB) / 633.051 (MB) / 695 (MB)
[Open3D DEBUG] Point weight / Estimated Area: 2.829664e-05 / 7.471727e-01
[Open3D DEBUG] #       Finalized tree: 0.0748339 (s), 474.375 (MB) / 633.051 (MB) / 695 (MB)
[Open3D DEBUG] #  Set FEM constraints: 0.157445 (s), 474.375 (MB) / 633.051 (MB) / 695 (MB)
[Open3D DEBUG] #Set point constraints: 0.0294471 (s), 474.375 (MB) / 633.051 (MB) / 695 (MB)
[Open3D DEBUG] Leaf Nodes / Active Nodes / Ghost Nodes: 497092 / 460824 / 107281
[Open3D DEBUG] Memory Usage: 474.375 MB
[Open3D DEBUG] # Linear system solved: 0.314828 (s), 474.375 (MB) / 633.051 (MB) / 695 (MB)
[Open3D DEBUG] Got average: 0.00745511 (s), 474.375 (MB) / 633.051 (MB) / 695 (MB)
[Open3D DEBUG] Iso-Value: 4.973715e-01 = 1.313309e+04 / 2.640500e+04
[Open3D DEBUG] #          Total Solve:       3.8 (s),     633.1 (MB)
geometry::TriangleMesh with 83232 points and 165338 triangles.
In [28]:
print('visualize densities')
densities = np.asarray(densities)
density_colors = plt.get_cmap('plasma')(
    (densities - densities.min()) / (densities.max() - densities.min()))
density_colors = density_colors[:, :3]
density_mesh = o3d.geometry.TriangleMesh()
density_mesh.vertices = mesh.vertices
density_mesh.triangles = mesh.triangles
density_mesh.triangle_normals = mesh.triangle_normals
density_mesh.vertex_colors = o3d.utility.Vector3dVector(density_colors)
o3d.visualization.draw_geometries([density_mesh],
                                  zoom=0.664,
                                  front=[-0.4761, -0.4698, -0.7434],
                                  lookat=[1.8900, 3.2596, 0.9284],
                                  up=[0.2304, -0.8825, 0.4101])

visualize densities
In [29]:
print('remove low density vertices')
vertices_to_remove = densities < np.quantile(densities, 0.01)
mesh.remove_vertices_by_mask(vertices_to_remove)
print(mesh)
o3d.visualization.draw_geometries([mesh],
                                  zoom=0.664,
                                  front=[-0.4761, -0.4698, -0.7434],
                                  lookat=[1.8900, 3.2596, 0.9284],
                                  up=[0.2304, -0.8825, 0.4101])

remove low density vertices
geometry::TriangleMesh with 82399 points and 163391 triangles.
In [30]:
# SAVE Poisson surface reconstruction (n=12), remove low density vertices
o3d.io.write_triangle_mesh("poisson_mesh.ply", mesh)
Out[30]:
True

Normal Estimation

In the examples above we assumed that the point cloud has normals that point outwards. However, not all point clouds already come with associated normals. Open3D can be used to estimate point cloud normals with estimate_normals, which locally fits a plane per 3D point to derive the normal. However, the estimated normals might not be consistently oriented. orient_normals_consistent_tangent_plane propagates the normal orientation using a minimum spanning tree.

In [31]:
gt_mesh = o3dtut.get_bunny_mesh()
pcd = gt_mesh.sample_points_poisson_disk(5000)
pcd.normals = o3d.utility.Vector3dVector(np.zeros(
    (1, 3)))  # invalidate existing normals

pcd.estimate_normals()
o3d.visualization.draw_geometries([pcd], point_show_normal=True)
In [32]:
gt_mesh = o3d.io.read_triangle_mesh("../../test_data/my test data/model_mesh.ply")
pcd = gt_mesh.sample_points_poisson_disk(5000)
pcd.normals = o3d.utility.Vector3dVector(np.zeros(
    (1, 3)))  # invalidate existing normals

pcd.estimate_normals()
o3d.visualization.draw_geometries([pcd], point_show_normal=True)
In [33]:
pcd = o3d.io.read_point_cloud("../../test_data/my test data/050.ply") 
pcd.normals = o3d.utility.Vector3dVector(np.zeros(
    (1, 3)))  # invalidate existing normals

pcd.estimate_normals()
o3d.visualization.draw_geometries([pcd], point_show_normal=True)
In [23]:
pcd.orient_normals_consistent_tangent_plane(100)
o3d.visualization.draw_geometries([pcd], point_show_normal=True)

---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-23-51071da5292c> in <module>
----> 1 pcd.orient_normals_consistent_tangent_plane(100)
      2 o3d.visualization.draw_geometries([pcd], point_show_normal=True)

AttributeError: 'open3d.open3d_pybind.geometry.PointCloud' object has no attribute 'orient_normals_consistent_tangent_plane'
In [ ]: